SQL and serde support#1154
Conversation
… serialisable & deserialisable
…`pub`lic functions necessary for other crates to implement asynchronous execution plans + execution of them
…lan is separated from run ; [src/lib/types.rs] Imbue `pub struct ExecutionPlan` with `steps_to_run` & `name` fields
|
@SamuelMarks just released rust info and git info |
|
question about diesel. why do we need it integrated to cargo-make? that feels like implementation details to something you will use outside of cargo-make. |
|
@sagiegurari It's annoying that Not even sure embedded types would solve this. The only obvious solution is to store struct records as JSON in a column of the SQL table. But what if I want an ExecutionPlan table? - And a Step table? - And a table joining the two that also includes State column, something like PS: This PR is almost ready for merge IMHO, just a little bit more rendering around the edges TODO. Just trying to get the full demo in place: 1 . TOML → saved somewhere (e.g., flat-file JSON or database) (these steps could for example be executed in parallel and on multiple machines) AND importantly I want to get this workflow supported without dramatically changing cargo-make. Just a little smattering of good stuff atop (error handling; use-as-library; serialising/deserialising & diesel derived types; and minor ergonomic improvements). cargo-make main use-case remains a Makefile alternative not a new Apache Airflow! |
…ib/runner.rs] Make `run_protected_flow` public ; [Cargo.toml] Use official `git_info` and `rust_info` now that my PRs have been merged ;
|
@SamuelMarks how about first you get some demo working for you than when you get a final understanding of what you need to make it work for you, revisit this PR and see the bare minimum needed for cargo-make to change to support it.
in general, i do'nt want to couple entire execution and structs to something very specific you are building. i want to enable extensions but not hard coupling. |
|
@sagiegurari Ok I've converted this to draft whilst I get it into a working state. Not long now [hopefully]! The main issue with only supporting serde seems to be representing Anyway watch this space! - Probably 24 to 48 hours away. |
WiP
Related PRs:
Took care to make the dependencies optional. I am preparing
cargo-maketo support #1110: asynchronous and distributed task execution. To support this, it is becoming necessary to store tasks and run task flows (execution plans). Whether that's as JSON (say for Redis or a flat-file solution), or as proper SQL tables (say for PostgreSQL or SQLite).